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ABSTRACT 



This report reviews work by the Department of Health and 
Human Services Office of Inspector General (OIG) , which evaluated the State 
Children's Health Insurance Program (SCHIP) , including whether children are 
being enrolled inappropriately in SCHIP rather than in Medicaid, and states' 
progress in reducing the number of uninsured children. This report asserts 
that OIG's assessment of appropriate enrollment would benefit from an 
expanded selection of states, and that while states' evaluations offered 
limited results, future OIG reviews may benefit from improved data sources. 
The report recommends that, in order to better inform the Congress on states’ 
efforts to implement SCHIP, the HHS inspector general: (1) expand the scope 

of the statutorily required periodic reviews to include all states with 
separate child health programs, including those with combination programs; 
and (2) consider using its general audit authority to explore whether issues 
of appropriate SCHIP enrollment also exist among states that have opted for 
Medicaid expansions under SCHIP, and should therefore be included in future 
OIG reviews. Comments from the OIG are appended. (EV) 
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Accountability * Integrity * Reliability 



United States General Accounting Office 
Washington, DC 20548 



March 29, 2002 

The Honorable Max Baucus 
Chairman 

The Honorable Charles E. Grassley 
Ranking Minority Member 
Committee on Finance 
United States Senate 

The Honorable W.J. “Billy” Tauzin 
Chairman 

The Honorable John D. Dingell 
Ranking Minority Member 
Committee on Energy and Commerce 
House of Representatives 

The Congress created the State Children’s Health Insurance Program 
(SCHIP) in 1997 to reduce the number of uninsured children in families 
with incomes that are too high to qualify for Medicaid. 1 For SCHIP, the 
Congress appropriated $40 billion over 10 years, with funds allocated 
annually to the 50 states, the District of Columbia, 2 and the U.S. 
commonwealths and territories. Financed jointly by the states and the 
federal government, SCHIP offers a strong incentive for states to 
participate by offering a higher federal matching rate — that is, the federal 
government pays a larger proportion of program expenditures — than the 
Medicaid program. 3 While this incentive encourages efforts to reduce the 
number of uninsured children through state participation in SCHIP, 
concerns existed that states might inappropriately enroll Medicaid-eligible 
children in SCHIP and thus obtain higher federal matching funds than 



'Medicaid is a federal-state program that provides health care coverage to certain 
categories of low-income adults and children: SCHIP was established as title XXI of the 
Social Security Act by P.L. 105-33 and is classified to 42 U.S.C. § 1397aa et seq. 

2 The District of Columbia is considered a state for purposes of this report. 

3 SCHIP offers an “enhanced” federal matching rate that is derived from a state's Medicaid 
rate. Each state’s match from SCHIP is equal to 70 percent of its Medicaid matching rate 
plus 30 percentage points, not to exceed a federal share of 85 percent. While the federal 
share of expenditures for Medicaid can range from 50 to 77 percent, federal shares of 
SCHIP expenditures can range from 65 to 84 percent. 
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allowed under Medicaid. 4 Inappropriate SCHIP enrollment also can affect 
what benefits are available for children because the SCHIP statute allows 
states to (1) expand their Medicaid programs, thus affording SCHIP- 
eligible children the same benefits and services that the state Medicaid 
program provides, (2) construct separate child health programs distinct 
from Medicaid that could provide more limited benefit packages and could 
include copayments that are generally not permitted for children in 
Medicaid, or (3) use a combination of both approaches. To address 
concerns regarding inappropriate enrollment, the SCHIP statute requires 
states to screen all SCHIP applicants for Medicaid eligibility and, if they 
are eligible, enroll them in Medicaid. 

Even with the requirement for Medicaid screening in place, concerns 
remained that children who were eligible for Medicaid might have been 
inappropriately enrolled in SCHIP. 5 Additionally, there was interest in 
assessing the progress states had made to reduce the number of uninsured 
children, including the extent to which states had met objectives and 
goals, which they established in their SCHIP programs. 6 In the Medicare, 
Medicaid and SCHIP Balanced Budget Refinement Act of 1999 (BBRA), the 
Congress directed the Department of Health and Human Services (HHS) 
Office of Inspector General (OIG) to conduct a series of studies on these 
issues. 7 BBRA specified that the OIG should review states that provide 
SCHIP coverage separately from their Medicaid programs. 

BBRA also directed that we review and report on the OIG’s work. In 
response, we assessed the OIG’s efforts to inform the Congress on (1) 
determining whether Medicaid-eligible children were improperly enrolled 
in SCHIP and (2) assessing states’ progress in reducing the number of 
uninsured children, including the progress states have made in meeting the 
objectives and goals initially established in their SCHIP programs. 



4 See U.S. General Accounting Office, Children’s Health Insurance Program: State 
Implementation Approaches Are Evolving, GAO/HEHS-99-65 (Washington, D.C.: May 14, 
1999). 

5 See H.R. Rep. No. 106-199, at 60 (1999). 

6 The SCHIP statute includes a provision requiring states, in establishing their programs, to 
specify strategic objectives and performance goals for providing child health assistance 
under SCHIP. See 42 U.S.C. §1397gg. 

7 BBRA amended the Social Security Act to require the HHS OIG to audit a sample of states 
beginning in fiscal year 2000 and every third fiscal year thereafter. 
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To examine these issues, we reviewed the OIG’s approach and 
methodology for selecting its sample of states for the first in its series of 
studies to evaluate states’ performance in screening SCHIP applicants for 
Medicaid eligibility and to assess states’ progress in reducing the number 
of low-income uninsured children. We examined the OIG’s findings in the 
context of other research, including our own work. 8 In some cases, we 
reviewed work released after the OIG’s studies were completed and 
published to determine the extent to which other research corroborated 
the OIG’s findings. Finally, we examined OIG’s recommendations to the 
Health Care Financing Administration (HCFA) and the Health Resources 
and Services Administration (HRSA), which jointly oversee SCHIP. 9 Our 
work was conducted from December 2001 through March 2002 in 
accordance with generally accepted government auditing standards. 



Results in Brief 



In responding to the mandate, the OIG published two reports, the first 
addressing whether Medicaid-eligible children were enrolled in SCHIP and 
the second assessing states’ progress in reducing the number of uninsured 
children. 10 The scope of the OIG studies included sampling 5 of the 13 
states that only enrolled children in separate child health programs during 
1999. The OIG concluded that Medicaid-eligible children were not being 
enrolled in SCHIP by the 13 states that administer separate child health 
programs. However, because of variations in the administration of state 
programs, generalizing from the findings in 5 states to the 13 states may 
not be appropriate. Furthermore, the issue of appropriate enrollment is 
not limited to states with completely separate child health programs, but 
also applies to those states with combination programs and Medicaid 
expansions, which also receive the higher SCHIP matching rate for state 
program expenditures. Because the scope of the study was limited to the 
13 states with separate SCHIP programs, the experience of other states — 
particularly the 13 states that operated SCHIP combination programs — 



8 See the related products listed at the end of this report. 

June 2001, the secretary of HHS announced that HCFA’s name would be changed to the 
Centers for Medicare and Medicaid Services. For this report, we will continue to refer to 
HCFA where our findings apply to the organizational structure and operations associated 
with that name. 

10 See Department of Health and Human Services, Office of Inspector General, State 
Children's Health Insurance Pivgram: Ensuring Medicaid Eligibles Are Not Enrolled in 
SCHIP , OEI-05-00-00241 (Washington, D.C.: Feb. 2001), and Department of Health and 
Human Services, Office of Inspector General, State Children’s Health Insurance Program: 
Assessment of State Evaluations Reports, OEI-05-00-00240 (Washington, D.C.: Feb. 2001). 
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was not addressed. Had the scope of review included the 13 additional 
states that offered separate child health programs under combination 
plans, the proportion of children represented would have increased from 
16.5 percent to 65 percent of SCHIP enrollees in 1999. Future OIG reviews 
that consider differences in enrollment practices across states and a wider 
universe of states could provide more information on the effectiveness of 
states’ efforts to ensure appropriate enrollment. 

Due to limitations the OIG identified in states’ SCHIP evaluations, the OIG 
was unable to conclude whether states were making progress in reducing 
the number of uninsured children and in meeting the objectives and goals 
they established in their SCHIP programs. For example, the OIG found 
that the 5 states it reviewed generally did not have sound methodologies to 
determine reductions in the number of uninsured children, in part because 
they did not always take into account other factors — such as changes in 
the economy or private insurance coverage — that also may affect the 
number of uninsured children. Furthermore, the OIG found that some 
states had set program goals without considering how they might be 
measured and that states’ staffs often lacked adequate evaluation skills. 
Based on its findings, the OIG made recommendations to HCFA and HRSA 
to help improve states’ ability to conduct more rigorous evaluations of 
their programs. Because of the limitations identified by the OIG, it may 
wish to look beyond states’ own evaluations and analyze other sources of 
analysis for its next review. Over time, other federal initiatives — such as 
improvements in state-level estimates of the number of uninsured — may 
help states to improve their measurement of progress under SCHIP. 

We are recommending that the HHS inspector general expand the scope of 
analysis to include a broader array of states to further inform the Congress 
on states’ progress in ensuring appropriate SCHIP enrollment. The OIG 
concurred with our recommendations, and provided general comments 
regarding approaches to designing future reviews. 


Background 


States provide health care coverage to low-income uninsured children 
largely through two federal-state programs — Medicaid and SCHIP. Since 
1965, Medicaid has financed health care coverage for certain categories of 
low-income individuals — over half of whom are children. To expand 
health coverage for children, the Congress created SCHIP in 1997 for 
children living in families whose incomes exceed the eligibility limits for 
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Medicaid. Although SCHIP is generally targeted at families with incomes at 
or below 200 percent of the federal poverty level, each state may set its 
own income eligibility limits within certain guidelines. 11 As of February 
2002, 16 states have created Medicaid expansion programs, 16 states have 
separate child health programs, and 19 states have combination Medicaid 
expansions and separate child health components. (See figure 1.) 



"in general, the SCHIP statute targets children in families with incomes at or below 200 
percent of the poverty level, which equates to $36,200 for a family of four in 2002. The 
statute allows a state to expand eligibility up to 50 percentage points above its Medicaid 
income eligibility standard in 1997. See 42 U.S.C. § 1397jJ(b)(l)(B)(ii)(I). 
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Figure 1: States’ Design Choices Under SCHIP, as of February 2002 




Separate child health program (16) 



Combination program (19) 



Note: Since the period reviewed by the OIG (1999), 8 states have altered their design choices under 
SCHIP. Seven states— Illinois, Indiana, Maryland, Mississippi, North Dakota, South Dakota, and 
Texas — have changed from Medicaid expansions to combination programs. West Virginia changed 
from a combination program to a separate child health program. 

Source: Centers for Medicare and Medicaid Services (CMS). 



SCHIP offers significant flexibility in program design and benefits 
provided by allowing states to use existing Medicaid structures or create 
child health programs that are separate from Medicaid. Medicaid 
expansions must follow Medicaid eligibility rules and cost-sharing 
requirements, which are generally not allowed for children. A Medicaid 
expansion also creates an entitlement by requiring a state to continue 
providing services to eligible children even when its SCHIP allotment is 
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exhausted. 12 In contrast, a state that chooses a separate child health 
program approach may introduce limited cost-sharing. Additionally, a 
state with a separate child health program under SCHIP may limit its own 
annual contribution, create waiting lists, or stop enrollment once the funds 
it budgeted for SCHIP are exhausted. States choosing combination 
programs take both approaches. For example, Connecticut’s combination 
SCHIP program has a limited Medicaid expansion — increasing eligibility 
for 17 to 18 year olds up to 185 percent of the federal poverty level. 
Additionally, the state created a separate child health program, which 
covers all children in families with incomes over 185 percent, up to 300 
percent of the federal poverty level. 

With regard to program benefits, the choices states make in designing 
SCHIP have important implications. For example, a state opting for a 
Medicaid expansion under SCHIP must provide the same benefits offered 
under its Medicaid program. These benefits are quite broad and include 
Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) 
services for most children. 13 EPSDT services are designed to target health 
conditions and problems for which children are at risk, including iron 
deficiency, obesity, lead poisoning, and dental disease. These services are 
also intended to detect and correct conditions that can hinder a child’s 
learning and development, such as vision and hearing problems. 14 In 
contrast, states opting for separate child health programs may depart from 
Medicaid requirements and provide benefits based on coverage standards 



12 However, states that expend their available SCHIP funds may then claim Medicaid 
matching rates for benefits and services provided under Medicaid expansions. 

13 EPSDT is optional for the medically needy population, a category of individuals who 
generally have too much income to qualify for Medicaid but have “spent down” their 
income by incurring medical and/or remedial care expenses. See 42 U.S.C. § 1396 
(a)(10)(C). 

14 For additional information on EPSDT, see U.S. General Accounting Office, Medicaid: 
Stronger Efforts Needed to Ensure Children’s Access to Health Screening Services, 
GAO-Ol-749 (Washington, D.C.: July 13, 2001). 
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in the SCHIP legislation. 16 SCHIP separate child health programs generally 
cover basic benefits, such as physician services, inpatient and outpatient 
hospital services, and laboratory and radiological services. Other benefits, 
such as prescription drugs and hearing, mental health, dental, and vision 
services, may be provided at the states’ discretion. States also may place 
limits on services provided and require cost-sharing, while Medicaid 
generally does not permit cost-sharing for children. 

In addition to having flexibility in program design and benefits offered, 
states participating in SCHIP have a larger proportion of their program 
expenditures paid by the federal government than for Medicaid. A state’s 
Medicaid program expenditures are matched by the federal government 
using a formula that is based on a state’s per capita income in relationship 
to the national average. Federal matching rates for SCHIP Eire 
“enhanced” — they are established under a formula that takes 70 percent of 
a state’s Medicaid matching rate and adds 30 percentage points, with an 
overall federal share that may not exceed 85 percent. 16 For 2001, federal 
shares of SCHIP expenditures ranged from 65 to 84 percent, with the 
national average federal share equaling about 72 percent. In contrast, 2001 
federal shares for Medicaid ranged from 50 to 77 percent of expenditures, 
with the national average at about 57 percent. The SCHIP statute requires 
states to screen all SCHIP applicants for Medicaid eligibility and, if they 
are eligible, enroll them in Medicaid. 17 



15 In prescribing a package of benefits, states with separate child health programs choose 
among four coverage standards. First, the benchmark standard provides coverage 
equivalent to that received by federal employees, state employees, or those enrolled in a 
state’s health maintenance organization with the largest insured commercial non-Medicaid 
enrollment. Second, the benchmark equivalent standard provides basic coverage for 
inpatient and outpatient hospital care; physicians’ surgical and medical services; laboratory 
and x-ray services; and well-baby and well-child care, including age-appropriate 
immunizations. Third, existing comprehensive state coverage includes benefit packages for 
state-operated children’s health insurance programs in Florida, New York, and 
Pennsylvania. Fourth, states may receive approval from the secretary of health and human 
services for benefit packages that provide appropriate coverage for low-income children 
but do not match the first three standards. 

16 For example, a state with the minimum 50-percent Medicaid match receives a 65-percent 
match under SCHIP. 

17 See 42 U.S.C. § 1397bb(b)(3). 



o 

ERIC 



Page 8 



12 



GAO-02-512 OIG Child Health Reviews 



OIG Studies 



BBRA included a mandate that the OIG conduct a study every 3 years, 
beginning in fiscal year 2000, to (1) determine the number, if any, of 
enrollees in SCHIP who are eligible for Medicaid and (2) assess states’ 
progress in reducing the number of uninsured low-income children, 
including progress in achieving the strategic objectives and performance 
goals in their SCHIP plans, which set forth how states intend to use their 
SCHIP funds to provide child health assistance. 18 

BBRA directed the OIG to review states with approved SCHIP programs 
that do not provide health benefits under Medicaid; 19 consequently, the 
OIG focused on the 15 states that in 1999 operated separate child health 
programs under SCHIP. 20 Of these 15 states, the OIG excluded 2 states — 
Washington and Wyoming — because the delayed start-up of their 
programs resulted in no enrollees in fiscal year 1999, the year that the OIG 
reviewed. From the remaining 13 states, the OIG used a two-stage 
sampling plan to select 5 states for review. The OIG first divided the 13 
states into two strata, selecting Pennsylvania separately as stratum I 
because it had a large number of children — 81,758 — enrolled in its 
program in fiscal year 1999. Enrollment across the remaining 12 states 
ranged from 1,019 in Montana to 57,300 in North Carolina. The OIG 
randomly selected 4 of the 12 states (North Carolina, Oregon, Utah, and 
Vermont) for inclusion in its study. (See table 1.) 



18 The OIG is charged with protecting the integrity of HHS programs, as well as the health 
and welfare of the beneficiaries of those programs. The OIG’s duties are carried out 
through a nationwide network of audits, investigations, inspections, and other mission- 
related functions. The OIG informs the secretary and the Congress of program and 
management problems and recommends legislative, regulatory, and operational 
approaches to correct them. The OIG may conduct its own evaluations or those mandated 
by the Congress. 

19 The BBRA mandate provides that “A state described in this [mandate] is a state with an 
approved state child health plan. . .that does not, as part of such plan, provide health 
benefits coverage under the State’s Medicaid program.” 42 U.S.C. § 1397hh(d)(2). 

20 The 15 states were Arizona, Colorado, Delaware, Georgia, Kansas, Montana, Nevada, 
North Carolina, Oregon, Pennsylvania, Utah, Vermont, Virginia, Washington, and Wyoming. 
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Table 1: OIG Sampling Framework for SCHIP Analysis, Fiscal Year 1999 



Stratum 


State 


SCHIP enrollment 


1 (Selected by OIG) 


Pennsylvania 


81,758 


II (Universe of states for purposes of 
random selection) 


Arizona 


26,807 




Colorado 


24,116 




Delaware 


2,433 




Georgia 


47,581 




Kansas 


14,443 




Montana 


1,019 




Nevada 


7,802 




North Carolina' 


57,300 




Oregon 0 


27,285 




Utah 0 


13,040 




Vermont 0 


2,055 




Virginia 


16,895 



‘State was randomly selected for the OIG’s review. 
Source: CMS enrollment figures for 1999. 



For the 5 sample states, the OIG reviewed a variety of documents the 
states submitted to HCFA, such as their SCHIP plans and SCHIP 
evaluation reports, which are states’ assessments of the effectiveness of 
their programs. 21 OIG staff conducted site visits and met with officials 
responsible for administering SCHIP in all 5 states. The OIG also randomly 
selected 100 active SCHIP case files from each of the 5 states in order to 
evaluate whether Medicaid-eligible children were incorrectly enrolled in 
SCHIP. The OIG did not verify accuracy and completeness of the state 
case files; rather, it focused on whether the information in each file 
supported the conclusion reached by the state. 



21 A state’s SCHIP evaluation was required to address several areas of analysis, including (1) 
the quality of health coverage provided, (2) its choices of health benefits coverage, (3) 
activities in coordinating SCHIP with other public and private programs, (4) changes in 
trends in the states that affect the provision of health insurance, and (5) recommendations 
for improving SCHIP. 
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OIG’s Assessment of 
Appropriate 
Enrollment Would 
Benefit From an 
Expanded Selection 
of States 



In determining whether Medicaid-eligible children were improperly 
enrolled in SCHIP, the OIG reported that, based on a sample of 5 states, 
SCHIP enrollees in the 13 states with separate child health programs were 
generally appropriately enrolled. However, because of variations in the 
administration of state programs, generalizing from the 5 states to the 13 
states may not be appropriate. In addition, focusing on only those states 
with separate SCHIP programs does not capture the experience of the 
majority of states or the meyority of SCHIP-enrolled children. Ensuring 
appropriate enrollment in SCHIP is important regardless of a state’s SCHIP 
design, because any child eligible for Medicaid that is incorrectly enrolled 
in SCHIP results in a state receiving a higher federal matching rate. 
Reviewing states, for example, that operate separate child health programs 
as part of a combination program would have increased the proportion of 
children under consideration from 16.5 percent to 65 percent of all SCHIP 
children enrolled in 1999, and thus provided more comprehensive 
information regarding states’ enrollment practices. 



Alternative Sampling 
Methodologies May More 
Fully Account for Variation 
among States 



To determine whether states were improperly enrolling Medicaid-eligible 
children in SCHIP, the OIG separated the 13 states with separate child 
health programs into two strata. The first stratum was the state of 
Pennsylvania, which the OIG intentionally selected because it had the 
most children enrolled in SCHIP among the 13 states. Four states were 
then randomly selected from the remaining 12 states. Among the 5 states it 
reviewed, the OIG identified only a few cases in which Medicaid-eligible 
children were inappropriately enrolled. 22 For example, it reported that 1 
state had a single case in which a Medicaid-eligible child was enrolled in 
SCHIP, while 2 other states had three and five such cases. The report also 
found that 2 states did not have any Medicaid-eligible children enrolled in 
SCHIP. The OIG concluded from these findings that most SCHIP enrollees 
were correctly enrolled in the 13 states administering separate child health 
programs. 



Variations in states’ enrollment practices, however, raise questions about 
the extent to which results from a sample of 5 states can be generalized to 
13 states. Had the OIG drawn its random sample of active SCHIP cases 
across the 13 states in its sampling universe, it would have been better 



“Based on a two-stage stratified-cluster sample, the OIG estimated that, at a 90-percent 
confidence level, from 97.6 to 99.6 percent of SCHIP enrollees were correctly enrolled in 
the 13 states administering separate child health programs in fiscal year 1999. 
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able to generalize its results. An OIG official told us that the office chose 
to analyze a sample of 5 states rather than all 13 states because of time and 
resource constraints. Recognizing that analyzing a pure random sample of 
cases across a large number of states may be too resource intensive, 
choosing a stratified sample of states may provide more information on 
the extent to which accurate enrollment may vary with different states’ 
practices. Even with a stratified sample, however, generalization to all 
states may be problematic. 

The OIG did select a stratified sample and chose one characteristic — size 
of a state’s SCHIP program— to develop two strata While dividing states in 
terms of size is potentially useful, additional distinctions may be important 
because program characteristics vary considerably from state to state. For 
example, states with differing administrative structures (New York uses 
health plans to determine eligibility and enroll eligible individuals, 
Colorado uses an enrollment contractor, and Oregon uses its Medicaid 
staff to determine SCHIP eligibility) could be grouped by certain 
characteristics for review. This could help determine whether such 
differences in administrative structures have a bearing on appropriate 
enrollment in SCHIP. 

To examine whether the OIG’s sampling approach reflected variations in 
states’ administrative structures, we categorized the 12 states in the 
second stratum based on whether they had the same program staff 
determine eligibility for both the SCHIP and Medicaid programs, which 
can help achieve consistency in eligibility decisions. We found that the 
random sample of 4 states did not include any states where different 
employees were responsible for determining SCHIP and Medicaid 
eligibility, thus raising concerns as to whether conclusions could be 
generalized. (See table 2.) 
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Table 2: Enrollment Practice of 12 States from Which the Random Sample Was 
Drawn 



Increasing the Number of 
States under Consideration 
Would Better Inform the 
Congress 



o BEST COPY AVAILABLE 

ERIC 



OIG stratum II 


State 


Who determines SCHIP and 
Medicaid eligibility — same 
staff or different staff? 


Randomly selected 


North Carolina 


Same 




Oregon 


Same 




Utah 


Same 




Vermont 


Same 


States not selected 


Arizona 


Different 




Colorado 


Different 




Delaware 


Same 




Georgia 


Different 




Kansas 


Same 




Montana 


Different 8 




Nevada 


Different 




Virginia 


Same 



“Montana generally uses a different staff to determine eligibility for each program; however, the 
state’s annual report notes that when children lose Medicaid coverage due to increases in family 
income, the Medicaid eligibility staff can enroll the children in SCHIP. In these circumstances, the 
same staff members would enroll children in both Medicaid and SCHIP. 

Source: SCHIP annual reports and state evaluations, 1999, 2000 and 2001. 



Because the scope of the study was limited to the 13 states with separate 
child health programs, the OIG examined 322,534, or 16.5 percent, of the 
approximately 2 million children enrolled in SCHIP in fiscal year 1999. A 
review that also included separate SCHIP programs in states that opted for 
a combination approach under SCHIP would have expanded the available 
universe to 26 states and to 65 percent of all SCHIP children enrolled in 
1999. 23 Moreover, using the OIG’s general audit authority, the scope of 
future reviews could include states with SCHIP Medicaid expansions, 
which would provide the Congress with more complete information on the 
extent to which states are enrolling low-income children in the 
appropriate programs. 24 If this approach had been used in 1999, 23 states 



^Some states have altered their design choices under SCHIP since 1999, which has resulted 
in more combination and separate child health programs in SCHIP (19 states and 16 states, 
respectively, as of February 2002). If the OIG were to include in its scope the experience of 
states with combination programs for the 2001 SCHIP enrollment of 4.6 million, it would 
have selected a sample from 35 states, or 74 percent of all children enrolled. 

^See 5 U.S.C. Appendix 3, § 4(a)(1). 
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and almost one-fourth of all children enrolled in SCHIP would have been 
added. (See table 3.) 



Table 3: States’ Design Choices and Percentage of Nationwide SCHIP Enrollment, 
Fiscal Year 1999 


SCHIP design 
choices 


States 


Percentage of total 
SCHIP enrollment 


Separate child 
health program 
(15 states) 


Arizona, Colorado, Delaware, Georgia, 
Kansas, Montana, Nevada, North Carolina, 
Oregon, Pennsylvania, Utah, Vermont, 
Virginia, Washington, and Wyoming. 


16.5 


Combination 
(13 states) 


Alabama, California, Connecticut, Florida, 
Iowa, Kentucky, Maine, Massachusetts, 
Michigan, New Hampshire, New Jersey, 
New York, and West Virginia. 


48.7 a 


Medicaid 
expansion 
(23 states) 


Alaska, Arkansas, District of Columbia, 
Hawaii, Idaho, Illinois, Indiana, Louisiana, 
Maryland, Minnesota, Mississippi, Missouri, 
Nebraska, New Mexico, North Dakota, Ohio, 
Oklahoma, Rhode Island, South Carolina, 
South Dakota, Tennessee, Texas, and 
Wisconsin. 


23.3 



’States with SCHIP combination programs have both a separate child health program and a Medicaid 
expansion component. The 48.7 percent cited in the table does not include the 11.1 percent of 
children who are enrolled in SCHIP Medicaid expansion components in these states. 

Source: CMS. 



While States’ 
Evaluations Offered 
Limited Results, 
Future OIG Reviews 
May Benefit from 
Improved Data 
Sources 



The OIG identified important limitations to states’ evaluations that made it 
unable to conclude whether states were making progress in reducing the 
number of uninsured children and in meeting the objectives and goals that 
they established under SCHIP. For example, the OIG found that states 
made inappropriate assumptions in reporting data about the relationship 
of SCHIP enrollment to the rates of uninsured, which undermined the 
credibility of states’ results, and that states often had poor baseline data 
against which to measure progress. The OIG also found that states set 
goals without considering how to evaluate progress, and that little 
emphasis was placed on evaluation by the states. As a result, the OIG 
made recommendations to both HCFA and HRSA on ways that the federal 
government could assist and guide states in making improvements in their 
analyses. While the initial OIG reviews were inconclusive due to 
weaknesses in states’ evaluations, future efforts may benefit from federal 
initiatives under way aimed at improving state-level data and analyses of 
SCHIP. These initiatives, however, may not have been in place long 
enough to benefit the OIG’s next review, since results are due in 2003. As a 
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result, the OIG may wish to select a different approach— such as 
identifying states with more rigorous practices in evaluation, or 
augmenting its review with other sources beyond those provided by the 
states. 



Weaknesses in States’ 
SCHIP Evaluations Limited 
the OIG’s Ability to 
Measure Progress 



The OIG identified limitations to the 5 states 1 SCHIP evaluations and thus 
was unable to draw conclusions about states' progress in reducing the 
number of uninsured children or meeting their stated objectives and goals. 
For example, the OIG cited concerns regarding the reliability of states’ 
reports of reductions in the number of uninsured, including inadequate 
data and evaluation practices. In cases in which states were unable to 
measure objectives that were established at the beginning of their SCHIP 
programs, their evaluations generally provided descriptive information on 
activities but did not assess the effect that such activities had on achieving 
specific goals. (See table 4.) For example, the OIG reported that none of 
the 5 states it reviewed attempted evaluations of their outreach programs 
or offered explanations of how such programs affected their measurable 
progress in enrollment or the number of uninsured children. 



Table 4: Limitations to Five States’ SCHIP Evaluations Identified by the OIG, 
February 2001 



Limitation 

Data problems and 
evaluation practices 
impaired evaluations 



State reports were 
descriptive, not 

evaluative 

Goals were set without 
evaluation in mind 
Evaluation was not 
considered a priority 
Staff members lacked 
evaluation skills and 
training 



Description 

• State-collected data were deficient or outdated. 

• State-level estimates based on national survey data were 
unreliable, particularly for smaller states. 

• States assumed that increased SCHIP enrollment meant 

reductions in uninsured. 

• Evaluations described activities without determining 
whether the activities were effective. 

• Information provided was qualitative and subjective. 

• Goals could not be measured. ~ ~ 

• Evaluation practices were not established. 

• Administrators were focused on implementing programs 

rather than evaluating their success. 

• SCHIP staff members were trained program administrators, 
but generally lacked thorough understanding of evaluation 
concepts and practices. 

• SCHIP staffs were small, making it unlikely that additional 

evaluation staff members would be hired. 



Source: HHS OIG, State Children's Health Insurance Program: Assessment of State Evaluations 
Reports, O El -05-00-00240 (Washington, D.C.: Feb. 2001). 



Of particular concern were limitations in measuring how well states are 
meeting the primary objective of the SCHIP program— reducing the 
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number of uninsured. As noted by the OIG, states — and other 
researchers — have been hampered by limited reliable state-level data 
regarding children’s insurance status. When SCHIP was enacted, estimates 
of the number of low-income uninsured children were derived from the 
annual health insurance supplement to the Current Population Survey 
(CPS), the only nationwide source of information on uninsured children by 
state. CPS is based on a nationally representative sample and is considered 
adequate to produce national estimates. 25 However, CPS data have well- 
recognized shortcomings, particularly with regard to state-level estimates, 
which can be unreliable and exhibit volatility from year to year because of 
small samples of uninsured low-income children, particularly in states 
with smaller populations. For example, using the 1994 through 1996 CPS 
data, estimates of the number of uninsured children in Delaware ranged 
from 12,000 to 32,000. In part because of these data limitations, some 
states — including 3 of the states sampled by the OIG — moved to special 
surveys or studies that were conducted locally in an effort to develop 
more precise estimates of the number of uninsured children. 

Despite efforts by states to better estimate the number of uninsured 
children, the OIG cited concerns regarding states’ analyses. For example, 
the OIG reported that some states estimated reductions in the number of 
uninsured children by subtracting the number of SCHIP enrollees from 
their original baseline estimates. However, such an approach does not 
ensure that increases in SCHIP lead to reductions in the number of 
uninsured because increases in SCHIP enrollment can result from children 
moving from private insurance coverage to public insurance under SCHIP, 
an effect known as “crowd-out.” Additionally, changing economic factors 
can further complicate assessments of a state’s progress in reducing the 
number of uninsured children. For example, a state may significantly 
increase enrollment in SCHIP but — because of declines in the economy 
and increased unemployment — continue to see an increase in the number 
of uninsured. Under these circumstances, “progress” in reducing the 
number of uninsured may be more difficult to identify. 

Based on its findings, the OIG recommended that HCFA identify a core set 
of evaluation measures that will enable all SCHIP states to provide useful 



26 CPS is a monthly survey of about 50,000 households. It is the primary source of 
information on the labor force characteristics of the U.S. population, and estimates 
obtained from CPS include employment, unemployment, earnings, and hours of work. 
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information. 26 It further recommended that HCFA and HRSA provide 
guidance and assistance to states in conducting useful evaluations of their 
programs. The OIG noted that SCHIP staffs would benefit from assistance 
and training regarding the type of data to collect and how to conduct 
evaluations. HCFA concurred with these recommendations and cited 
efforts under way to improve states’ evaluations of their SCHIP 
programs. 27 



Subsequent OIG Reviews Several federal efforts are under way that should help improve states’ data 

May Benefit from Efforts sources and their evaluations of the extent to which their SCHIP programs 

to Improve Bata Sources 3X6 reducing the number of uninsured children. If implemented on a timely 

basis, efforts such as the following should help inform the OIG’s 
subsequent evaluations. 



o The Congress appropriated $10 million each year beginning in fiscal year 
2000 to increase the sample size of CPS. Beginning in 2001, larger sample 
sizes are being phased into CPS, which should help improve the accuracy 
of state-level CPS estimates of uninsured children. 28 
o CMS is working with states to develop consistent performance measures 
for SCHIP, with a focus on ensuring appropriate methodology and 
consistency of data. 

° As a condition of their state SCHIP plans, some states are required to 
assess whether the SCHIP program is “crowding out” private health 
insurance in their states. These studies could help assess the extent to 
which SCHIP is drawing its enrollment from uninsured children — or from 
children who were previously insured. 

° BBRA requires HHS to conduct an evaluation of SCHIP to determine the 
effectiveness of the program and to provide information to guide future 
federal and state policy. To comply with BBRA, HHS plans a series of 
reports addressing a variety of mqjor topic areas, ranging from program 
design to access and utilization; the first report is expected in spring 2002. 
HHS plans to use multiple research strategies, including case studies, 
surveys, and focus groups, to address questions of interest. 



26 HHS OIG, State Children’s Health Insurance Program: Assessment of State Evaluation 
Reports, OEI-05-00-00240 (Washington, D.C.: Feb. 2001). 

27 HRSA did not comment on the recommendations made by the OIG. 

23 Data from the expanded sample are expected to appear in all CPS-based reports 
beginning in 2002. 
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As the OIG continues to analyze states’ progress in SCHIP, its future 
reviews are likely to benefit from improvements in state-level estimates of 
the number of uninsured children and evaluations of program 
implementation. Moreover, improvements in states’ analyses and available 
data should help the OIG identify and address areas in need of additional 
review. However, to the extent that these improvements are not in place 
by the time the OIG undertakes its second analysis due in 2003, it may 
benefit from expanding its scope of work to identify and assess states with 
more rigorous analyses. The OIG may also wish to review other sources 
that have assisted states in making evaluation improvements. For 
example, while some states have received private grant funds to help with 
SCHIP enrollment, they have also received technical assistance for the 
purpose of conducting evaluations on the success of their enrollment 
strategies. 29 Other states have paired with universities or research 
organizations to improve their information on the uninsured. By also 
drawing on the experience of states with strong evaluations or data 
sources, the OIG will be better able to identify approaches that could 
further strengthen federal and states’ approaches and inform the Congress 
on progress in implementing SCHIP. 



Conclusions 



Through its periodic evaluations of states’ efforts to ensure appropriate 
SCHIP enrollment and to reduce the number of uninsured children, the 
OIG is in a position to provide objective information to the Congress and 
others about the program’s operation and success. To better capture the 
experience of all states, regardless of the design of their SCHIP programs, 
the OIG should expand its scope beyond the 13 states in its first review to 
also include states that operate separate child health programs within 
SCHIP combination programs and consider including Medicaid expansion 
programs as well. This would provide a broader base for understanding 
how well states are screening for Medicaid eligibility and identifying issues 
related to reducing the number of uninsured children. Such an expansion 
of scope may also help identify states with more rigorous evaluations of 
their SCHIP programs, and thus provide information on effective 
approaches to SCHIP evaluation as well as more complete information for 
the Congress. 



2 °For example, the Robert Wood Johnson Foundation funds initiatives that assist states and 
others to expand health insurance coverage. Among other things, the foundation has 
published an evaluation tool to guide policymakers throughout the evaluation process. 
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Recommendations to 
the HHS Inspector 
General 



In order to better inform the Congress on states’ efforts to implement 
SCHIP, we recommend that the HHS inspector general 

expand the scope of the statutorily required periodic reviews to include all 
states with separate child health programs, including those with 
combination programs, and 

consider using its general audit authority to explore whether issues of 
appropriate SCHIP enrollment also exist among states that have opted for 
Medicaid expansions under SCHIP, and should therefore be included in 
future OIG reviews. 



Agency Comments 
and Our Evaluation 



We provided the inspector general of HHS an opportunity to comment on 
a draft of this report. In its comments, the OIG concurred with our 
recommendations, and agreed that expanding the scope of its inspections 
to include combination programs that include separate child health 
programs would give a greater breadth of information. It also agreed that 
including SCHIP Medicaid expansions would broaden the perspective and 
present more conclusive information regarding the status of states’ SCHIP 
programs. 

The OIG also provided general comments regarding its approach and 
possible approaches to designing future reviews. For example, the OIG 
stated that it would consider including differing state processes as a factor 
in its next sample design. The OIG also noted the importance of focusing 
on states’ measurement of their own program performance. We agree with 
the OIG that properly conducted state evaluations serve a vital function 
and we believe that continued review of these efforts by the OIG is an 
important contribution to better understanding states’ progress under 
SCHIP. In response to the OIG’s oral and written comments, we revised 
the report to better clarify the scope of the BBRA mandate. The full text of 
the OIG’s written comments is reprinted in appendix I. 



We are sending copies of this report to the inspector general of the 
Department of Health and Human Services and other interested parties. 
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We will also make copies available to others on request. If you or your 
staffs have questions about this report, please contact me on (202) 512- 
7118 or Carolyn Yocom at (202) 512-4931. JoAnn Martinez-Shriver also 
made contributions to this report. 

fa . t — 

Kathryn G. Allen 
Director, Health Care — Medicaid 
and Private Health Insurance Issues 
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Appendix I: Comments from the Department 
of Health and Human Services’ Office of 
Inspector General 
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Appendix I: Comments from the Department of Health and Human Services’ Office of 
Inspector General 



COMMENTS OF THE DEPARTMENT OF HEALTH AND HUMAN SERVICES ON 
THE U.S. GENERAL ACCOUNTING OFFICE’S DRAFT REPORT, “CHILDREN’S 
HEALTH INSURANCE: INSPECTOR GENERAL REVIEWS SHOULD BE EXPANDED 
TO FURTHER INFORM CONGRESS” 



The Office of Inspector General (OIG) for Health and Human Services (HHS) appreciates the 
opportunity to review GAO’s draff report Children $ Health Insurance: Inspector General 
Reviews Should Be Expanded To Further Inform Congress, GAO-02 -5 12. This report offers 
salient observations regarding two studies issued by the OIG in February 200 1 . The OIG reports, 
State Children s Health Insurance Program - Ensuring Medicaid Eligibles are not Enrolled in 
SCHIP, OEI-05-00-00241 and State Children 's Health Insurance Program - Assessment of State 
Evaluations Reports OEI-05-00-00240 were produced to meet the mandate specified in the 
Balance Budget Refinement Act of 1999 (BBRA) for the OIG to prepare such reports and to 
offer States, the Center for Medicare and Medicaid, and Congress information regarding certain 
facets of the SCHIP program. We concur with GAO’s recommendations. 

General Comments 

As pointed out by GAO, the sample for the 200 1 reports was stratified by size. GAO has noted 
that other sampling methods could have been used and suggested that a more effective technique 
would have been to stratify the sample based upon administrative features used by the States, 
such as the separation of SCHIP and Medicaid staffs for making eligibility determinations. No 
sampling method guarantees unbiased results. We believe that our sampling method was 
appropriate. It is possible that the differences in enrollment processes between the various States 
could affect the results of this study but we cannot be certain of this. Nevertheless there may be 
merit in GAO’s suggestion. When we select the sample for the 2003 studies, we will consider 
the State processes as a factor in the sample design. 

The GAO suggested that we consider data sources other than those available from States in our 
future reviews. The OIG will bear in mind the suggestions made regarding our future 
measurement of the reliability and validity of States’ evaluation efforts. However, we utilized 
the most complete and accurate data available at the time of our review. As GAO pointed out, it 
is unlikely that the Federal efforts to improve States’ data sources will be completed and States 
may still be experiencing problems with the reliability and validity of the baseline data utilized as 
part of their evaluations. 

While we will consider the other data sources available to measure State performance, the OIG 
maintains that it will remain important to continue focusing on what States are doing to measure 
their own program performance. We believe that future reports should evaluate what progress 
has been made by States related to establishing measurable goals and training staff so that they 
can perform the evaluation process. Properly conducted State evaluations serve a vital function. 
They not only inform the Secretary of Health and Human Services of the progress being made to 
meet the States’ performance goals but they also serve as program management tools for the 
States. 
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Appendix I: Comments from the Department of Health and Human Services’ Office of 
Inspector General 



GAO Recommendation 

We recommend that the HHS Inspector General expand the scope of the statutorily required 
periodic reviews to include all States with separate child health programs, including those with 
combination programs. 

Department Comment 

We agree that expanding the scope of our inspections to include combination programs would 
give a greater breadth of information. The OIG is presently in the process of developing the FY 
2003 Work Plan, and will consider expanding the scope of our work to include combination 
programs in our future work and data analysis. 

GAO Recommendation 

We recommend that the HHS Inspector General consider using its general audit authority to 
explore whether issues of appropriate SC HIP enrollment also exist among States that have opted 
for a Medicaid expansion under SCHIP, and should therefore be included in future OIG reviews. 

Department Comment 

As with the previous recommendation, the OIG agrees that the inclusion of such programs in our 
analysis would broaden our perspective and present more conclusive information regarding the 
status of SCHIP programs. Inclusion of these programs will be considered in our future work. 
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GAO’s Mission 


The General Accounting Office, the investigative arm of Congress, exists to 
support Congress in meeting its constitutional responsibilities and to help 
improve the performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates federal 
programs and policies; and provides analyses, recommendations, and other 
assistance to help Congress make informed oversight, policy, and funding 
decisions. GAO’s commitment to good government is reflected in its core values 
of accountability, integrity, and reliability. 
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and phrases. You can print these documents in their entirety, including charts and 
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Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 
daily. The list contains links to the full-text document files. To have GAO e-mail 
this list to you every afternoon, go to www.gao.gov and select "Subscribe to daily 
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